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ABSTRACT 

at a public medical school to provide individualized continuing education to 
participants and to foster closer ties between academic health center faculty 
and community practitioners who serve as preceptors for health professions 
students. Various methods have been used to evaluate the program, with 
focuses on program implementation and processes and program outcomes for 
participants. Specific sources of data were program management systems, 
end-of-day surveys of visiting clinicians, periodic interviews with visiting 
clinicians, follow-up surveys of clinicians who have completed the program, 
and surveys of faculty hosts. Findings indicated the following: (1) the 

program enrolled 103 clinicians; (2) visiting clinicians identified 139 
specific learning needs, the most frequently identified areas being diabetes , 
cardiology, pain, adult obesity, sports medicine, dermatology, and computing 
in medical practice; (3) the program had financial and political costs and 
constraints; (4) administrative aspects of the program were satisfactory both 
for visiting clinicians and host faculty; (5) program participants engaged in 
587 half- or full-day sessions in 116 different areas since the program began 
and earned 2,884 hours of continuing medical education credit; and (6) the 
program appeared to be a vehicle for building and strengthening collegial 
relationships between program participants and the faculty members who host 
them. (YLB) 
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Abstract 

We describe the evaluation of a continuing medical education program by using multiple 
methods that have evolved over five years in response to the various needs of stakeholders. 
Program management systems, surveys, and interviews were used to gather data about an 
individualized continuing education program for community-based primary care physicians and 
nurse practitioners. We focused on program implementation, processes, and outcomes. The 
findings have proven useful in making program decisions and determining program 
effectiveness. The desire for rigor in using social science methodology often conflicts with 
shifting situational realities and the need to be utilization-focused. 



Introduction: Perspectives and Purpose 



“They say there was method to his madness. Perhaps so. It is easier to select a method for 
madness than a single best method for evaluation, though attempting the latter is an excellent 
way of achieving the former” (Patton, 1997, p. 241). 



The Visiting Clinician Program was established in 1996 at a public medical school to provide 



individualized continuing education to participants and to foster closer ties between academic 



health center faculty and community practitioners from across the state who serve as preceptors 
for health professions students. Few tangible rewards are given to these preceptors for their 
teaching, which represents an essential service to the university in the provision of thousands of 
hours of instruction to students each year. The Visiting Clinician Program, with its attendant 
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continuing education credits, is provided free of charge, thereby rewarding preceptors while 
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helping to ensure that the knowledge and skills that they teach students are current. The program 
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was modeled after a similar initiative at West Virginia University; no other programs like them 
are known to exist (Curtis et al., 1999). 

The program also was designed to counter some of the criticisms that have been leveled at 
continuing health professions education in recent years and respond to suggestions for new 
approaches to increase the effectiveness of continuing education in changing and improving 
practice behavior. Moore and colleagues (1994) point out that the contemporary emphasis on 
continuous quality improvement in health care, with the aim of improving patient outcomes, 
highlights the need for continuing education that is demonstrably effective in helping 
practitioners change their practice to provide better care. Davis and colleagues (1992, 1995, 
1999) conducted a number of studies to assess the impact of various approaches to continuing 
medical education. They concluded that sessions that provide opportunities for active learning 
and practicing related to specific learning needs can effect change in professional practice, while 
traditional didactic sessions (e.g., lectures) do not. 

These findings are congruent with three key findings from recent research on learning (Bransford 
et al., 2000). First, learners come with preconceptions that must be elicited and engaged; 
consequently, interaction between teacher and learner is essential. Second, to develop 
competence, learners need to gain factual knowledge, develop an understanding of facts and 
ideas within a conceptual framework, and organize knowledge in ways that facilitate retrieval 
and application. This suggests the importance of opportunities to both acquire and apply 
knowledge and skills in situations that mirror actual practice. Third, a metacognitive approach, in 
which learners monitor their own understanding of specific subject areas, helps learners take 
control of their own learning. Continuing education programs therefore should help learners 
analyze their learning needs and assess their progress and the extent to which they have achieved 
their learning objectives. 

The Visiting Clinician Program exemplifies many of these characteristics of effective continuing 
education. At enrollment, each participant analyzes his or her individual educational needs and 
develops learning objectives based on those needs. Program staff then arrange a series of one- 
day visits for the participant with appropriate faculty members, with whom the visitor works 1:1 
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to achieve the set objectives. Learning activities include observation of clinical care; discussions 
with faculty, residents, and patients about specific clinical problems and questions; referrals by 
faculty to additional sources of information; and attendance at special or regularly scheduled 
departmental conferences or presentations. The program maintains an extensive menu of 
options, each with a designated faculty host, which has grown larger each year and currently 
includes approximately 200 offerings across multiple specialties, up from 38 offerings in the first 
year of the program. Faculty members do not receive compensation for their service as visiting 
clinician hosts. Typically, a visiting clinician will choose two learning activities for each day- 
long visit, one for the morning and another for the afternoon, but some visits involve a full day in 
the chosen area. During the initial years of the program, each participant attended seven one-day 
visits over a two-year period; in the last two years, the program has consisted of four one-day 
visits over a one-year period. This change was instituted to permit more practitioners to enroll. 

Potential participants are nominated by faculty and administrators from the departments of 
Family Medicine, Medicine, Obstetrics and Gynecology, and Pediatrics. For three years, 
program participants also included nurse practitioners who were nominated by the School of 
Nursing. Since its inception, the program has served 103 primary care clinicians from across the 
state. An advisory panel composed of representatives from clinical departments, hospital 
administration, and medical school administration (and for three years, nursing school 
administration) meets on an annual basis to review program accomplishments and provide advice 
on matters such as program improvement strategies, evaluation, and funding. 

The Visiting Clinician Program has an array of stakeholders, including visiting clinicians, faculty 
hosts, advisory panel members, several funders, medical and nursing school administrators, 
hospital administrators, and program administrators and staff. Conducting situationally 
responsive evaluation involves reacting to stakeholders’ interests and needs and adapting 
evaluation questions and designs as conditions change (Patton, 1997). Evaluation efforts have 
had to reflect the multiple, varying, shifting, and sometimes contradictory interests, 
responsibilities, and information needs of these groups. For example, all of the stakeholders are 
interested in program effectiveness and outcomes. In addition, visiting clinicians are interested 
in program processes (e.g., convenience, ease), funders and administrators care about program 
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efficiency and public relations, and faculty hosts want to know about the program’s impact on 
their clinical responsibilities. In general, program evaluation has focused on (1) program 
implementation and processes and (2) program outcomes for participants. The specific questions 
that have guided the evaluation are as follows: 

How effective are program implementation and processes? 

• How many practitioners does the program serve and what are their characteristics? 

• What do the participants identify as their educational needs? 

• How do participants determine their educational needs? 

• What are the costs of the program? 

• How satisfactory are the administrative aspects of the program (scheduling, etc.) for 
participants and host faculty? 

What are the outcomes for participants and faculty? 

• What educational benefits accrue to the visiting clinicians? 

• What changes do participants make in their professional practice as a result of the 
program? 

• How much continuing education credit do participants earn? 

• What are the benefits of the program for the faculty hosts? 

• What relationships develop between participants and the academic health center? 

Method 

We have used a variety of methods to answer the questions listed above, including written 
surveys, interviews, and analysis of program management data. Many of the strategies that we 
developed at the beginning of the program were altered in various ways as the program grew and 
as various stakeholders and staff members came and went. Other methods were added 
incrementally. Data gathering and analysis continue as we proceed with the program. Reported 
here are findings from the program’s first five and one-half years. 

Specific sources of data were as follows: 
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• Program management systems. We maintain databases that include demographic 
information about program participants, including name, address, specialty, gender, age, 
and whether located in a Health Professional Shortage Area (HPSA). We also maintain 
information about clinicians’ learning goals and activities and extent of continuing 
education credit, and we have information about the departments, offices, clinics, and 
host faculty available to the visiting clinicians. In addition, minutes of advisory 
committee meetings and notes from meetings with representatives from funding agencies 
provide data about funding priorities, budgetary constraints, and political issues. 

• End-of-day surveys of visiting clinicians. These written surveys are administered at the 
end of each day’s visit and ask the clinician to rate the level of perceived educational 
benefit of the day’s activities, using a 4-point scale (no benefit to significant benefit). 

The clinician rates the level of benefit in the following areas: confirmation of existing 
skills, acquisition of new knowledge and skills, improved proficiency in diagnosis and 
treatment, and exposure to different clinical settings. The participant also is asked 
whether he or she achieved the educational objective that had been set for the day. 

Finally, if the clinician had an opportunity to co-teach with the faculty host, the clinician 
is asked to describe these teaching activities and indicate extent of satisfaction with them 
(not at all to completely). A total of 312 surveys have been completed; however, the 
wording and scaling of items have been refined over time. 

• Periodic interviews with visiting clinicians. During specified visits, a program staff or 
faculty member meets with the visiting clinician to ask open-ended questions focusing on 
the process by which the visitor decided on learning objectives and activities, degree of 
success in meeting those objectives, future learning needs and plans, and suggestions for 
program improvement. The questions asked during the interview have changed over time 
to focus less on satisfaction with the program and more on learning objectives and needs. 
A total of 142 interviews have been documented. 

• Follow-up surveys of clinicians who have completed the program. Six months after 
program completion, a written survey is sent to the clinician. Open-ended questions ask 
clinicians to describe specific changes they made in patient care practice as a result of 
participation in the program, the outcomes of those changes for patients, changes in 
teaching practice, types of relationships maintained with academic health center faculty 
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or staff, and suggestions for improvement in the program. Of the 37 participants from the 
first three cohorts (1996-98, 1997-99, 1998-00) who have completed the program, 28 
have submitted follow-up surveys. 

• Surveys of faculty hosts. After each visit, the faculty host is sent a written survey that 
asks the host to indicate level of agreement with statements about the visitor’s integration 
into the clinic or workplace, contributions to students’ or residents’ learning, provision of 
a community perspective to the clinic, whether the host gained useful information from 
the visitor, and whether the program is an effective way to introduce community 
practitioners to his or her specialized services. A scale of 1 to 5 (strongly disagree to 
strongly agree) is used. In addition, open-ended questions ask about the host’s 
preparations for the visit, referrals of the visitor to other resources, and his or her interest 
in serving again as host. The use of these surveys was instituted several years after the 
start of the program so that we could better answer hospital administrators’ questions 
about faculty perceptions of the program’s utility and impact (e.g., potential disruption of 
faculty and clinic schedules). Faculty hosts have completed a total of 218 surveys. 

Analysis entailed compilation of program information from the database, computation of means 
and standard deviations for the scaled survey items, and qualitative analysis of narrative data 
through repeated reading of the data and identifying categories and themes. In some cases, 
analysis of data across all five years of the program required interpreting and compiling data 
from several different versions of the same instrument. In such cases, judgments were made 
jointly by several staff members. 

Results 

Program implementation and processes 

The program has enrolled 103 clinicians, including those enrolled during the 2001-02 academic 
year: 89 MDs or DOs and 14 nurse practitioners; 92 Caucasians, 7 African-Americans, 2 Asians, 
1 Hispanic, and 1 Native American; 72 males and 31 females; 31 from Family Medicine, 31 
from Internal Medicine, 15 from Ob/Gyn, and 26 from Pediatrics. Forty-eight visiting clinicians 
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(47%) practice in Health Professional Shortage Areas. These characteristics are similar to those 
of health care providers statewide. 

Visiting clinicians identified 139 specific learning needs across many specialties. The areas most 
frequently identified included diabetes, cardiology, pain, adult obesity, sports medicine, 
dermatology, and computing in medical practice. Typically, visiting clinicians determined 
learning needs by considering the problems encountered most frequently in practice, identifying 
new knowledge or skills that they wanted to acquire, reflecting on weaknesses or areas in which 
they felt uncomfortable, identifying long-term interests, and, in a few cases, seeing what was 
available on the days they could attend the program. 

The program has both financial and political costs and constraints. The annual budget has 
decreased by approximately half as the program has become better established, procedures more 
routine, and extraneous elements eliminated (e.g., stipends for visitors), but maintaining financial 
support for the program remains an ongoing task. Political issues have resulted in shifts in 
sources of funding as well as in program offerings. Initially, primary funding was provided by 
the health care system, but an overtaxed budget resulted in the withdrawal of that support after 
the first four years. Additional sources of funds at various times in the life of the program have 
been the School of Nursing and the state Area Health Education Centers (AHEC) Program. 

Nurse practitioners were included in the program in its third, fourth, and fifth years; however, the 
School of Nursing then withdrew its support. The program currently is funded through a small 
grant from a pharmaceutical company and in-kind support by the medical school. With regard to 
influences on program offerings, when the program was begun, some faculty and department 
chairs feared that participants would learn skills that would result in their no longer needing to 
refer patients to the academic health center. These faculty and chairs therefore refused to offer 
their clinics as learning sites. This problem has been eliminated by changes in departmental 
leadership over time as well as program evaluation data showing that program participants 
become more comfortable about referring their patients to the faculty whom they meet through 
the program. 
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Administrative aspects of the program in general have been satisfactory both for visiting 
clinicians and host faculty. Visiting clinicians routinely commend program staff for their work 
in scheduling visits, providing information, communicating with participants, and so on. 
Suggestions for improvement have been implemented whenever feasible, e.g., providing a 
clearer delineation of the visitor’s learning goals to the faculty hosts; however, problems such as 
confusing hospital layout and disorganized clinic scheduling were out of the control of the 
program. 

Outcomes for participants and faculty 

Program participants have engaged in 587 half or full-day sessions in 116 different areas since 
the program began and have earned a total of 2,884 hours of continuing medical education credit. 
The most frequently attended areas include diabetes, cardiology, computing, urgent care, pain, 
and infectious disease. While most participants attended areas related to their initially-defined 
learning objectives, some shifts occurred as participants’ needs or interests changed over time. 
Visitors reported that the program provided moderate to significant benefit in confirming 
existing skills, improving proficiency in diagnosis and treatment, acquiring knowledge, and 
providing exposure to various clinical settings. Participants reported marginal benefit in the 
acquisition of new technical skills, most likely because visitors are not permitted to engage in 
hand-on treatment of patients. Ninety percent of the responses to the question of whether the 
educational objectives for the day had been met were affirmative. 

Of the 28 who completed the six-month follow-up survey, 24 (86%) reported having made 
specific changes in their practice as a result of what they had learned as visiting clinicians. 
Respondents described the following: 

• 33 changes in approaches to treatment and management in areas such as prescribing, 
improved clinical procedures/techniques, greater initiative in treatment, new management 
strategies 

• 16 changes in diagnosis, such as new screening techniques or evaluation protocols, improved 
interpretation of studies, more appropriate use of tests 

• 8 changes in referring behavior, e.g., more frequent or appropriate referrals 
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• 6 changes in administrative aspects of practice, such as internet use, conversion to electronic 
records, use of flow sheets 

• 3 changes in patient education, such as the use of new materials and techniques 

Faculty hosts agreed that visitors integrated smoothly into the hosts’ workplaces, added a 
community practice perspective to the clinic, contributed to the learning experiences of students 
and residents, and provided useful information to the hosts. They reported gaining useful 
information from the interaction and found the program effective in introducing clinicians to 
their specialized services. Two hundred seven of the 212 (98%) responses to the question of 
whether the faculty member would host a visiting clinician again in the future were in the 
affirmative. Three faculty hosts, however, commented that the presence of the visitor slowed the 
work of the clinic. 

Visiting clinicians reported that they had maintained contacts with colleagues they had met 
through the program. These contacts included calling for consults and referring patients. One 
visitor went back to a faculty host as a patient. Participants also reported that they feel more 
comfortable referring patients to physicians whom they have met in person and to a facility with 
which they have become familiar. 

Discussion 

The Visiting Clinician Program has proven to be a valuable and effective continuing medical 
education strategy for community-based primary care physicians who serve as preceptors for 
medical students statewide. Our experience with the program confirms that tailoring active 
learning opportunities to individual learning needs is both feasible and effective in improving 
practice. Participants generally defined their learning needs based on the needs of their practice 
and gaps in their knowledge and skills. Further, the program appears to be a vehicle for building 
and strengthening collegial relationships between program participants and the faculty members 
who serve as their hosts. 
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Patton (1997) suggests that evaluations should be judged by their utility and actual use. The 
variety of methods we have used to evaluate the program have provided information useful both 
in continuous improvement of program processes and in judging the effect of the program on 
practice. In our case, meeting the information needs of multiple stakeholders has required a 
diverse array of evaluation methods and sources of data. Further, meeting the changing needs of 
changing sets of stakeholders has required us to continually revise and refine our methods, in 
some cases resulting in inconsistencies that are troublesome at best when trying to judge the 
value of the information generated and compile that information over a five-year period. 
Resource constraints also have precluded the use of certain strategies, such as chart review, to 
determine changes in practice behavior. We designed our six-month follow-up survey to be the 
next best approach to gathering these data. Asking participants to describe very specific changes 
that were related to their specific learning objectives and goals we felt would yield information 
as reliable as possible, given that it was generated through self-report. 

Despite these challenges, evaluation of the Visiting Clinician Program has yielded information 
that has been useful to most if not all of our stakeholders. Even though the program has been 
successful in meeting its intended purposes, the original funder decided to withdraw funding as 
part of more global cutbacks; however, documentation of outcomes and processes has prompted 
others to provide support. Evaluation data also have been useful in monitoring the relevance of 
the program to the visiting clinicians and correcting any problems that affect the faculty hosts. 

Program evaluation is necessarily utilitarian in focus, with an emphasis on providing information 
and conclusions that will be useful to decision-makers and reflective of the perspectives of 
multiple stakeholders. The characteristics of successful evaluation are utility, feasibility, 
propriety, and accuracy (Stufflebeam, 1980), and such evaluation requires situational 
responsiveness, methodological flexibility, multiple evaluator roles, political sophistication, and 
creativity (Patton, 1997). Patton holds that program evaluation is fundamentally different from 
research both in the purpose of data collection and standards forjudging quality: research aims to 
discover new knowledge, test theories, establish truth, and generalize, while evaluation is 
intended to inform decisions, clarify options, identify improvements, and provide programmatic 
information in the context of place, values, and politics. Program evaluation therefore 
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emphasizes systematic data collection rather than pure application of social science 
methodology. Evaluators may use research methods to gather information, but they may also use 
management information system data, program monitoring statistics, or other forms of 
systematic information that are not research-oriented. For educational researchers engaged in 
program evaluation, the desire for rigor in using social science methodology often conflicts with 
shifting situational realities and the need to be utilization-focused. 
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